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Title: Imaging of buried structures 

The invention relates to a method of obtaining an image of buried 
structures in an object, in particular to imaging structures such as vascular 
structures in biological tissue by means of selective combining information 
derived from tissue images in the visible range and the infra-red range. 
5 ' A method describing such is disclosed in the international 

application WOOl 15597 by the same inventor. It has been found, that . 
obtaining a sufficiently clear image is difficult due to various problems. One 
problem is, that light emanating from buried, in particular deeper parts of the 
object is often much weaker than the Ught that is reflected directly by the 

10 surface of the object. In practice, this means that a separation of the specularly 
reflected fight and light emerging firom deeper parts of the object may be 
needed in order to identify underlying structures. 

While separating these two types of light, for instance by a known 
method of using polarized Ught and using the fact that specularly reflected 

15 light keeps its polarization direction, so that it can be filtered out by means of 
a polarizing filter, a substantial amount of the light is lost that is originating 
from the lower parts of the object, thus resulting in a loss of image brightness 
and resolution. This invites to the use of powerful light sources in order to 
receive sufficient hght from the lower parts in the process of separating the 

20 two parts. However, especially in the area of imaging structures in live objects, 
there is a maximum amount of light that may be irradiated on the object. 

WOO 1/50955 shows a reflective arrangement where specular light is 
filtered by a polarizator. The image of an underlying structure is combined 
with a visual image to present a single image. However, it has been found that 

25 straightforward combination of these images offers problems, in particular, 

since certain artefacts occur that are inconvenient to reliably detect and locate 
an underlying structure. 
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US2001/0037811 shows a probe for determining a circumference of a 
finger. Additionally, the joint is transilluminated for inspecting arthritis 
inflammation firom a scattering analysis. This arrangement cannot be used for 
visually imaging body parts with a fireedom comparable to the human eye. 
5 The invention has as an object to provide an imaging technique that 

does not suffer firom the afore described problems and that is able to provide an 
enhanced image of the underlying structure. Moreover, the invention has as an 
object to provide an imaging enhancement technique to enable a person to 
combine visual information and information of buried objects in one image. 

10 To achieve these and other goals, in one aspect, the invention offers 

a method according to the features of claim 1. In another aspect, the invention 
offers a method according to the features of claim 14. 

In particular, by providing a camera for imaging visual and infrared- 
images; providing a bounded light source for partly irradiating said object by 

15 infi^ared light; partly irradiating said object by said bounded Ught source; 
imaging a non-irradiated area of said object by said camera to image said 

« 

buried structure; and combining said buried structure image with a visual 
image of said object, infrared Ught incident on the image that is originating 
fi'om a direct reflection is spatially filtered out of the image. The remaining 

20 partial image hence does not suffer from saturation effects due to direct 

illumination of specularly reflected light. In a preferred embodiment, a full 
image is provided by varying said partial irradiation in time so as to provide a 
full image by subsequent combining of said partial images. 

In a further preferred embodiment, said partial image is obtained by 

25 scanning a light beam over said object. In addition or alternatively, said 
partial image is obtained by subsequently irradiating said object by 
predetermined patterns. One particularly preferred embodiment comprises 
obtaining said partial image by alternatingly irradiating said object by a 
predetermined complementary patterns. For instance, in an embodiment said 
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patterns may be matrix-patterns, line patterns, dot patterns, concentric or 
circular patterns. 

Further, preferably said object is irradiated only at predetermined 
positions that are spaced apart. By spacing the irradiation area and the 
5 light detection area, deeper parts of the buried structure may be enhanced. 

By alternatively illuminating said object, a full image may be 
provided, and wherein all areas of the object are irradiated in a time- 
dependent manner. 

The invention further offers particular benefits while using a 

10 CMOS-camera, since these camera's have a high degree of decorrelation of 
adjacent pixels. Thus, the effect of "blooming" is prevented, so that there is a 
high contrast between the directly reflected area (that is discarded) and the 
remaining area which receives diffuse light originating from deeper layers. 

Furthermore, the invention is preferably used while aligning said 

15 infrared Ught source with a visual Ught source; providing a first edge analysis 
of said infrared image; providing a second edge analysis of said visual image; 
comparing said first and second edge analysis; and discarding edges in said 
infrared image that are also detected in said second image. This offers the 
benefit of obtaining a "normal" visual image (as seen by the human eye), that 

20 is enhanced by identifying the underlying structure (detected using infrared 
light) within the visual image while discarding false edges formed by 
shadowing or aliasing artifacts (e.g. shadows or reflections from a scalpel or 
needle). For instance, for surgical purposes, the method offers a convenient 
tool for deciding an optimal entry point in the object, for instance for cutting 

25 tissue or the like. 

Preferably, said edge-detection is performed by a gradient analysis 
of said first image. The invention offers a convenient embodiment when said 
images are provided stereoscopicaUy. Furthermore, said first image may be 
spectrally analysed, and wherein said spectral analysis is projected into said 

30 second image. Furthermore, said spectral analysis may comprise a pulsatihty 
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analysis and/or a hart beat frequency analysis and/or respiratory frequency 
analysis. Such analysis thus offers a convenient non-contact tool for measuring 
body parameters of interest. Under "pulsatility analysis" is understood at least 
a determination of pulsating parts in the object of interest. 
5 The invention is further related to a method of enhancing imaging of buried 
structures in an object, comprising: as defined in claim 14. Furthermore, the 
invention is related to a system as defined in claim 16. 

Further features and benefits will become apparent from the figures. 
10 In the figures: 

Figure 1 shows a schematic view of a scanning irradiating method of 
an object according to the invention; 

Figure 2 shows a schematic view of foton-migration in hve tissue 
due to diffusion; 

15 Figure 3 shows an illumination method by illuminating an 

overlapping pattern on the object; 

Figure 4 shows an illumination method by illuminating a circular 
patterns on the object; 

Figure 5 shows an illumination method by illuminating a grid of 
20 patterns on the object; 

Figure 6 shows a testing arrangement for testing the apparatus 
according to the invention using one visible wavelength and two infirared 
wavelengths; 

Figure 7 shows a series of analysis steps for processing and routing 
25 of data acquired from the arrangement of Figure 6; 

Figure 8 shows a spectral analysis of the pulsatile components in 
said image for light of three diSerent wavelengths; 

Figure 9 shows a simple vizor-like construction comprising the 
apparatus of the invention; 
30 Figure 10 shows the vizor-like construction of Figure 9 in use; 
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Figure 11 shows the VIS and NIR images collected by the eyepiece of 

Figure 9; 

Figure 12 illustrates the steps according to the method of the 
invention; and 

5 Figure 13 shows a further embodiment of the inventive system 

including an infrared emitting puncture tool. 



Figure 1 shows an alternative to the above described polarization 
filtering method. This method comprises dynamic complementary 
10 lighting/scanning of alternating patterned image sections. This method does 
not require the use of polarization filters. It is based on the fact that photons 
entering biological tissue will strongly scatter within the tissue which partly 
results in backscattering. 

Furthermore the viewed area is divided into parallel linear areas , 
15 which we will call "line sections". These line sections can be divided into even 
and uneven line sections 1, 2 respectively. 

Using a camera that has good anti-blooming specifications and allows 

« 

the read-out of freely selectable rectangular pixel regions we then can acquire 
image information in a special sequence. 

20 During a certain period the bounded light sovirce will light all even line 

sections 1 and the camera will acquire image information from all uneven line 
sections 2. Here, the term bounded encompasses light that is bounded 
spatially, so that an illuminated object comprises, in a direction of view which 
is also a direction of illumination, non-illuminated areas which are not reached 

25 by the bounded Ught source. Such bounding can be typically reached by 
focusing, coUimating or shielding the fight source. Also various other fight 
sources, such as laser fights (e.g. in combination with a holographic grating) 
and LEDs, can produce boxmded fights. During the next period the uneven fine 
sections 2 are lighted and the camera wiU acquire image information from the 

30 even line sections. This can either be done with a line camera that scans the 
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entire tissue or with a normal camera that scans all even lines simiiltaneously 
and during the next period all uneven Unes. 

In Figure 2 is illustrated how Ught diffused within the tissue can be 
used to image deeper parts of the object. Light enters the object at one position 
5 1 and leaves the object at another position 2. From the figure it becomes clear 
that hght entering at further distanced positions can collect information £rom 
parts deeper within the object. By applying a variety of illumination patterns 
as will be further described with reference to Figure 3 - Figure 5, illumination 
firom "within" the object can be achieved, thus imaging deeper parts of the 
10 object. 

To this end, in Figure 3, instead of even and uneven parallel lines 
shown in Figure 1, alternate spatiaUy shifted crossed hne patterns can be used 
as lighting pattern whilst image acquisition occurs within the areas between 
the lines. It is also possible to first project an "even" dot pattern on the cross 

15 point of line pattern 1 and then project an "uneven dot pattern on the cross 
sections of line pattern 2. Also, in Figure 4 discrete concentric circular areas 
can be used. Satisfying results were obtained by a simple test of circular 
irradiation geometry disclosed in Figure 4. A 6mm thick shoe of pink foam (3M 
ethafoam) was laid upon the shielding pipe. On top of this foam slice a plastic 

20 office clamp was placed, with its' white plastic paper retaining slice laying 
across it. Finally a second shoe of pink foam was laid on top. Outside the 
camera's field of view, a ring hght injected photons of 660 nm, 810 nm and 940 
nm into the foam, perpendicular to the foam surface. 

25 Figure 5 shows another embodiment, wherein said object is irradiated 

only at predetermined positions that are spaced apart. First areas 3 indicated 
with horizontal hnes are irradiated in a first period; second areas 4 with 
vertical lines are irradiated in a second period. Such a spaced apart 
configuration is able to show deeper parts of the structure. By varying the 

30 spacing, lower and deeper parts of the object may be scanned. 
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Figvire 6 shows a testing arrangement, where an embodiment of the 
inventive method was tested using known markers such as an SP02 pulse 
oximeter, an ECG recording devise and a respiratory frequency monitor. The 
5 signals were recorded and sampled using the steps indicated in Figure 7. This 
leads to a pixel by pixel time-analysis of intensity variation. The frames were 
sampled at a lOOHz Sample rate and the recorded respiration, ECG and 
plethismographic pulse output were compared. The outcome is illustrated for a 
variety of wavelengths in Figure 8. It is clearly shown how well the measured 
10 variation of the camera matches with the other pulse signals. 

A special configuration with added value is based upon two cameras 
(e.g. CCD or CMOS monochromatic or multiband ) positioned at a certain 
distance from each other (e.g. eye-to-eye distance) thus stereoscopically 
viewing the same object (e.g. biological tissue), a dual channel electronic image 
15 processing device and two display devices placed in front of both eyes. 

Between each camera and the viewed object an additional optical system 
(maybe combining 2 channels) may be placed (e.g., a dual channel microscope, 
endoscope, colposcope, etc.) It is possible to incorporate a simple vizor-like 
construction 5 (see Figure 9) so that the device can be either put in front of the 
20 eye or be positioned out of the viewing angle to allow normal sight. 

Figure 10 shows as an example the stereoscopic eyepiece 5 of Figure 9 
in use. For the eyepiece, a color camera can be apphed with a bayer color filter 
pattern of which all filter colors are highly transparent in the NIR-range. Also 
25 a multi-layered camera chip (Patent WO 02/27804 to Foveon Inc. or earher 
patents like 4,238,760 to Carr) or the approach in our previous patent WO 01 
15597 Al can be applied. Preferably, the images in both spectral ranges match 
pixel to pixel. 

The surface of the viewed biological tissue region 6 is irradiated at an 
30 obhque angle from two opposite sides by two light sources 7. These two sides 
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need not to be aligned with the Left/Right axis of the eyepiece 5, but can be 
swiveled around in a plane perpendicular to the central axis of the joined 
cameras. 

The light sources 7 can be controlled to independently send out 
5 broadband white light in the visible wavelength range (VIS; 400 - TSOnm) or 
narrowband light in the near in&ared range (NIR; e.g. 920 nm) as well as in 
both ranges (VIS & NIR). The Ught sources 7 are carefvilly constructed so that 
the geometrical beam profiles of VIS and NIR are aligned resulting in identical 
shadowing in the VIS and NIR images. As an alternative also a surgical stereo 

10 microscope or an endoscopic dual channel camera (as used in stereoscopic robot 
surgery) with two side fiber light guides can be used to collect the images. 

Figure 1 1 shows the result of the VIS (Figure 1 1 A) and NIR (Figure 1 1 
B) images such as collected by the eyepiece 5 illustrated in Figure 10. Since 
NIR and VIS beams are matched, shadows produced by irregular shapes at the 

15 tissue surface (e.g. skin structure, skin folds, molds, etc.) will also match in 

both wavelength ranges. In the embodiment disclosed, the beams are oriented 
at a small angle with respect to the tissue region 6. As a result, skin folds etc. 
will produce sharp-edged shadows. Shiny areas that produce reflections and/or 
saturated pixels (marked as a star) will also match in both wavelength ranges. 

20 Due to the small angle, objects 8 that are brought towards the tissue surface 6 
(e.g. needles, scalpels, probes, etc.) will produce two separate shadows. These 
shadows will meet and typically form a "V* pattern 9 when an object touches 
the surface. If e.g. a needle punctures the surface, then the needle tip 10 will 
quickly disappear firom the VIS image (Figure 11 A). On the NIR image 

25 (Figure 11 B) however, the tip 10 will remain visible within the tissue 6 (see 
figure). 

On the VIS image (Figure 11 A), only very superficial blood vessels will 
be visible, especially if the texture of the vessel deforms the skin. On the NIR 
image, however, blood vessels will be visible much better (even blood vessels at 
30 a depth of a few millimeters below the surface). Since skin pigment (melanin) 
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has a very low absorbance within the NIR region this good NIR visibility will 
also be the case with dark skinned persons (also molds will lose their dark 
color) - 

After obtaining the NIR and VIS images in steps 11 and 12 
5 respectively, a gradient edge enhancement step is appUed for both spectral 
bands as will be illustrated with reference to Figiure 16 (steps 13 and 14). 
Image VIS- J and NIR- J are obtained by filtering with a smoothing filter to 
suppress noise from camera image NIR and VIS. In the current 
implementation this is an averaging filter. From each smoothed images VIS-J 
10 and NIR-J, two directional difference images Ix and ly are calculated by means 
of a gradient filter. Currently this it performed with a Prewitt filter. 



Next edge gradient images VIS-G and NIR-G are made by yj(lx^ -^fy^)- 

Then VIS-G and NIR-G are clipped and normalized to 1 to obtain VIS-G' and 
NIR-G' (Steps 15 and 16). Finally the images are complemented to 1-G' (Steps 
15 17 and 18). 

For a given tissue region a VIS image and a pixel to pixel matching NIR 
image are acquired. 

The images from both spectral bands are subjected to an identical edge 
enhancement algorithm (see above), producing normalized NIR-edge and VIS- 
20 edge grayscale images so that edges are encoded black (0 = black and 1 = 
white; note that this is the negative of normal edge enhanced results). 




Now corresponding edges in the NIR and VIS image are determined 
(Step 19) according to a pixel by pixel criterion: 



25 



NIR-edge-pixel < NIR_Treshold AND VIS-edge-pixel < VIS_Treshold 
(Thresholds are both software configurable) 



wo 2005/079662 PCT/NL2005/000108 

10 



By combining information from both spectral regions, the pixel co- 
ordinates that match both requirements are identified as superficial artifacts 
and are discarded by setting the pixel vadue to 1 in the corrected NIR edge 
image (Step 20). 

5 By doing the same in the raw NIR image after normahzing (Step 21) (no 

edge enhancement) a corrected normalized NIR image (Step 22) is defined by 
discarding saturated image areas. 

To this end, the locations of all saturated pixels within the raw NIR 
image (e.g. 256 for an 8-bit image) are identified. For all pixels within the 1st 
10 NIR-mask that are satiirated or a direct neighbors of a saturated pixel, the 
edges are discarded by setting the pixel value to 1 (and thus completely 
ignoring edge information) is filled in, resulting in a second NIR-mask (step 
23). 

False edges resulting from saturated pixels are thus removed and the 
15 2nd NIR-mask now contains the boundaries of subcutaneous blood vessels (and 
eventual other buried contrasting structures). 

This approach can be varied by including second, third etc. neighboring 

pixels. 

In a preferred mode, the raw IR image is used to provide "filled in" blood 
20 vessels; for example, by multiplying the 2nd NIR-mask with the 2nd corrected 
NIR image (step 24). The amount of fill-in colorization can be varied with a 
user adjustable exponent N : 



2nd NIR-mask x (1st corrected NIR image) ^ = final enhancement 

25 mask 



Finally, this final enhancement mask now is multiplied with the 
luminosity component of the raw VIS image (which preferably is a color image) 
in step 25. In this way only contrast information that truly originates from 
30 below the tissue surface is projected into the VIS image. If we look at the 
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previous drawings with a needle inserted in tissue it is clear that these criteria 
will lead to superimposing only the buried needle tip upon the VIS image. 
Shadows, wrinkles, reflections and the edges of molds will not be enhanced. 
The detection of purely superficial artifacts thus can be further 
5 improved when instead of taking all colors of the visible region into account, 
only the Blue spectral range is used for the VIS edge enhancement input. This 
is done in order to suppress the visibihty of blood vessels. This effect of blue 
light is because in the visible range, the reflectivity of vessels for blue hght is 
the nearest to skin (the maximum visible vessel contrast Ues outside the blue 

10 range). As another optional refineiiient the final enhancement mask may be 
selectively multiplied with e.g. only the Red and/or Green portion of the visible 
image (instead of the luminosity signal of the total visible color image). 

When an interesting tissue portion is spotted, the user can switch to a 
mode that alters the image capturing sequence and allows the collection of an 

15 additional (second) NIR image within a selected region of interest (ROI). This 
ROI is virtually illuminated by photon injection just outside the ROI by means 
of EITHER a LED-array in contact with the tissue OR a pattern of laser dots 
or laser lines projected onto the tissue fi:om a distance. During acquisition of 
this second NIR image the VIS and first NIR lighting are off. 

20 In this way information obtained using two types of lighting geometries 

for the same wavelength range can be combined. This can be done with or 
without also using VIS and flashing needle tip lighting as will be illustrated 
with reference to Figure 13. 

In Figure 13 a NIR-laser or NIR-LED is plugged onto a needle or 

25 catheter 8. Via a sterile window light is projected through or alongside the 
needle bore. In this way the needle tip 10 illuminates the vessel interior wall 
26 when the needle is in the vessel (see fig 13B). If the needle punctures the 
distal vessel wall (see fig 13C) a drastic change in illumination is seen. The 
Hght source clipped to the needle may be synchronized with the cameras. A 

30 flashing tip illumination may be beneficial. 
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Although in the preceding the invention has been further illustrated 
with reference to the drawings and the description it will be clear that the 
invention is not limited thereto and that these embodiments are only disclosed 
for the pvirpose of illustration. Variations and modifications thereto are within 
5 the scope of the invention as defined in the annexed claims. 



